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ABSTRACT 

Revolutionary advancements have been observed in e-learning technologies though an 
amalgamated evaluation methodology for new generation e-learning content 
development tools is not available. The evaluation of educational software for online use 
must consider its usability and as well as its pedagogic effectiveness. This study is a first 
step towards the definition of criteria for evaluating e-learning tools. A preliminary user 
study involving a group of pre-service instructional designers, observed during their 
interaction with e-learning tools, is reported. Throughout the study, specific usability 
attributes of these e-learning tools were identified. Participants were assigned to rate 
the importance of functional and pedagogical competences proposed during the criteria 
development phase. The findings of the study revealed 31 evaluation criteria under the 
headings of technical, media, and assessment competences. Among the groups of 
benchmarks proposed and rated by the users, assessment was considered as the most 
important one while technical and media features were even.The following step was 
actual implemention of the usability criteria into evaluation of fifteen leading software 
used in e-learning across the world. Mostly, tools were observed as having limitations in 
terms of capabilities. Comparing to the other software. Captivate, Softchalk, and Lectora 
were regarded as outstanding tools by the participants. Following the discussion on the 
limitations of the study, some implications for further research were proposed. 
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INTRODUCTION 

Last two decades have witnessed the transformative effect of Internet on the design and 
development of e-learning environments which refers to the use of the internet and 
computer-based technologies to facilitate teaching and learning (Horton, 2006; Ruiz, 
Mintzer & Leipzig, 2006). Becoming very important in fields where access to learning 
materials needs to be brought about effectively and efficiently, e-learning is firmly 
embedded in many of the current educational theories. To examplify, it is widely 
recognized that learning is a social process (Vygotsky, 1978; Wenger, 1999) and Fowler 
and Mayes (2000) explain how learning relationships can encourage the 
conceptualization and re-conceptualization cycle which facilitates deep understanding. 
Kandies and Stern (1999) have asserted that web-enhanced learning improves 
instruction and course management and offers numerous pedagogical benefits for 
learners helping them become more active and self-directed learners. As with any other 
forms of learning, the strength of e-learning heavily relies on its delivery method and the 
instructional way by which specific media utilized. For this reason, the tools implemented 
into e-learning modules have to be pedagogically skillful in triggering an interactive, 
autonomous, and constructive learning climate. 


336 



The e-learning domain is getting over its early stages of development and classic tutorial 
types courseware are replaced to interactive and LMS friendly content development 
options. The development of content for e-learning environments includes the need of 
large teams, time constraints, and costs caused the emergence of rapid e-learning 
development tools. Nowadays, e-learning vendors have produced a number of e-learning 
authoring tools for creating learning objects for incorporation into learning or course 
management systems. These broadly split as follows: 'PowerPoint Plugin Authoring Tools' 
which are very easy to use as most people are familiar with PowerPoint. 'Desktop 
Authoring Tools' which are installed on your desktop and are generally more complex 
than the PowerPoint tools but they give you more control over the style and interactions. 
'Server-Based Tools' which are hosted on a server and are typically accessed via a web 
browser over the internet. In their book, 'E-learning Tools and Technologies', William 
and Katherine Horton (2003) mention the roles of e-learning development tools in course 
and website authoring, testing and assessment, media editors, and content converters. 
Merrill (2000) specifies a pedagogic architecture for such tools and names it as 'learning- 
oriented ID tool' which is one that has built-in instructional strategies which are based 
on the scientifically verified principles of instruction. 

These tools, apart from developing multimedia training materials, simplify to prepare and 
publish course content to more than one learning management system (LMS). The 
software allows the instructional designer or the content developers to generate 
storyboards and support a variety of media and file types, such as text, graphics, video, 
and audio. Most include assessment and test creation features. On the other hand, since 
there is a shortage of comprehensive manuals or guidelines developed for adoption of 
these tools, it may take a long time for the users to handle them efficiently. Relatively, in 
many cases the instructional designer is not a capable programmer, and often the 
proficient programmer is not an instructional designer. Hence, attempts to adopt and 
utilize e-learning tools may turn into a mind-confusing and laborious activity. 

Evaluation methods and curricula have become very important in a climate where there 
is increasing concern with assessing and maintaining quality (Jones et al. 1999). The 
adoption of e-learning tools deserves a profound attention and educational stakeholders 
require appropriate guidelines as well as effective evaluation methodologies to 
implement usable content development software (Zaharias, Vasslopoulou & 
Poulymenakou, 2002). However, a well-documented and widely accepted evaluation 
methodology of e-learning tools does not yet exist (Ardito et al. 2006). The current study 
attempts to respond a lack of particular guide for evaluating the usability of e-learning 
content development tools. 

Usability Evaluation of E-learning Tools 

The term usability was originally derived from the term 'user friendly' (Folmer et al. 
2002). Usability is considered as the most important aspect of any interactive software 
for educational settings (Zaharias et al. 2002). The ISO 9241 Standard (1988) defines 
usability as 'the extent to which a product can be used by specified users to achieve 
specified goals with effectiveness, efficiency and satisfaction in a specified context of 
use'. Since, usability plays a significant role towards the success of e-learning software, 
an e-learning environment or tool should be evaluated considering both its usability as 
software and its didactic effectiveness (Ardito et al. 2004; Ardito et al. 2006; Costabile et 
al. 2005). In other words, a systematic evaluation should concentrate on the platform 
and the educational modules (Tervakari, 2002). 

Moreover, the usability of e-learning environments has been defined as the extent to 
which an application is learnable and allows users to accomplish specified goals 
efficiently, effectively, and with a high degree of satisfaction. 
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In other words, if an e-learning tool is not usable enough, it prevents student's learning: 
the learners would not spend more time learning how to use the software rather than 
learning the contents. Evaluating usability is now considered an essential part of the 
system development process and a variety of methods and has been developed to 
support the human factors professional in this work. 

Silius and Tervakari (2003) say that it is important to evaluate the pedagogical design of 
e-learning systems. Notess (2001) asserts that usability testing needs additional 
consideration in the light of the web-based learning environments, such as learner 
satisfaction with the learning content, learner perception of the applicability of the 
content, learner enjoyment of the learning experience, and actual learning, measured via 
tests. 

The term "pedagogical usability" is used, in this paper, to denote whether the tools 
support various learners to learn in various learning contexts according to selected 
pedagogical objectives (Tervakari, 2002). According to Silius and Tervakari (2003) the 
pedagogical usability can be divided into three main categories as support for; 
organization of the teaching and studying, learning and tutoring processes as well as the 
achievement of learning, the development of learning skills. 

Various usability evaluation methods exist (Dix, Finlay, Abowd & Beale, 2004) of which 
some of the most well-known are predictive evaluation, heuristic evaluation, naturalistic 
observation, user-based methods such as questionnaires and interviews, and usability 
testing (Preece, Rogers, & Sharp, 2007). In broad terms it is worth making the following 
distinctions between evaluation methods as user-based: where a sample of the intended 
users try to use the application, expert-based: where a usability expert makes an 
assessment of the application, and model-based: where an expert employs formal 
methods to predict one or more criteria of user performance. Testing an application with 
a sample of users is generally considered as the most reliable and valid usability 
evaluation method (Dillon, 2001). In a user-based evaluation, test subjects are required 
to perform a set of previously defined tasks. Depending on the primary focus of the 
evaluator, the users' success at completing the tasks and their speed of performance are 
used to clarify the usability level. After the tasks are completed, users are often asked to 
provide data on likes and dislikes through a survey or interview, or may be asked to view 
with the evaluator part of their own performance on video and to describe in more detail 
their performance and perceptions of the application (Partala & Kallinen, 2012). 

The evaluation of software prior to its use, which, typically occurs when teachers are 
either planning lessons or making purchasing decisions (Squires, & Preece, 1996). The 
number of studies devoted to identify usability issues of e-learning systems and software 
is not large (Storey, Philipps, Maczewski & Wang, 2002). Moreover, it is often the case 
that the proposed criteria are only vaguely stated, so that an actual measurement is left 
to subjective interpretation and implementation. This paper has conducted work in the 
evaluation of various e-learning content development tools. The work presented in this 
paper is mostly focused on defining the usability aspects of e-learning software by 
soliciting the perceptions of prospective instructional designers as a group of users. The 
goal of this research is to reveal the results of an evaluation study in which various e- 
learning development tools were compared in terms of their pedagogic usability features. 
This comparison has been done in terms of technological competences, skills needed, and 
e-learning standards (SCORM). 

Twenty-eight independent evaluators have provided required information to compare the 
tools. Later these independent comparisons were analyzed to come up a consensus. The 
following research questions were investigated: 
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1. What are user-based usability benchmarks of e-learning software? 

2. What are users' evaluations toward usability guidelines of e-learning software? 

3. What are users' evaluations regarding various e-learning software? 

METHODS 

Participants 

The current study involved 28 senior students of a Multimedia Design, Development and 
Evaluation (MDDE) class at the Kirikkale University in Turkey. This course is provided to 
sophomore students attending computer education and instructional technologies 
departments of faculty of educations. They participated in the experiment as part of their 
credits for the MDDE course. The course curriculum involves authoring systems for PCs, 
courseware development phases, and interface design principles, imbedding multimedia 
into the software, user interaction, feedback systems, navigation, and multimedia 
evaluation. All participants had a basic knowledge of usability of interactive systems and 
of usability evaluation techniques, because they had some previous experiences of 
evaluating software systems. 

Procedure 

As a preliminary users study, a group of prospective instructional designers were 
observed during their interaction with an e-learning software in a real situation. The 
participants were provided with a list of the 15 leading e-learning content development 
tools and asked to develop an instructional module through the specific software they 
chose. The tools, identified through a review of literature, focused in the current study 
were; Softchalk, Content Point, Outstart Trainer, Tutor Author NG, Toolbook, Turbo 
Demo, Viewlet Builder, Ignite 4, Ready Go Web, Camtasia Studio, Lectora Inspire, 
Articulate, Mos SOLO, Course Lab, and Captivate. 

Before the experiment process began, all participants were presented with a short 
demonstration of the tools to be evaluated. Each tool is reviewed by a group of students 
who have regularly used it to build instruction. Each group, consisting of three people, 
participated in their specific evaluation sessions in which the usability technique adapted 
to the context of e-learning was illustrated. The experiment in the current study 
consisted of 12 weeks. During the first six weeks, participants elaborated on designing 
and categorizing an evaluation criteria used to rate the tools. In the meantime, in order 
to be exposed to details of the software, learners were also expected to design and 
develop an instructional module exemplifying the basic features of that particular 
software without assigning them any content themes. The participants were not limited 
on the content type they were required to develop through the software. The last six 
weeks of the semester were devoted to actual evaluation of the e-learning systems based 
on the guidelines (criteria) developed within the previous stage. 

Instrument 

The data were collected through an evaluation form developed by the participants of the 
study. The development phase of the evaluation form was also included in the study as a 
research issue. The aim to develop the evaluation form was to purport the significance 
levels of the educational usability benchmarks of e-learning content development tools' 
from the perspectives of prospective instructional designers. The evaluation form was 
also utilized to determine how participants rated specific software in terms of basic 
benchmarks emerged from the evaluation phase. The parameters included in the 
evaluation form were identified by the students through a semester long course focusing 
on multimedia design, development and evaluation provided to pre-service teachers 
attending computer education and instructional technologies departments of Faculties of 
Education in Turkey. 


339 


To identify items for possible inclusion in the evaluation form, participants were asked 
about their experiential learnings. Additionally, a number of web and instructional design 
guidelines have been reviewed by the students, as well as a number of usability 
evaluation heuristics, checklists, and questionnaires. More specifically, an extensive 
review of prior studies referring to e-learning and usability was conducted by the 
researcher and shared with the participants (Tervakari, 2002; Trinchero, 2004 cited from 
Ardito et al. 2006). On the evaluation form each question is accompanied by rating scale 
including "unimportant (1), of little importance (2), moderately important (3), important 
(4), and very important (5)". Evaluators are asked to rate the package of questions 
connected to each criterion e.g. media. The questions have been modified so that the 
evaluator does not have to think to which degree a particular criterion has been reached. 
The evaluator answers to simple questions like "is it possible for a user to change the 
font size of the web page by using browser settings?" The Cronbach's Values for the 
evaluation form were calculated as follows; technical competences (0.86), media 
competences (0.81), and assessment competences (0.89). 

ANALYSIS 

This chapter will provide a guideline rated in terms of importance by the prospective 
instructional designers. The data gathered were analyzed through descriptive statistics 
via SPSS 17.0 software. Means and standard deviations were provided within the tables. 
Table 1, including 14 criteria, depicts the participants' perceptions toward the importance 
levels of technical competences of e-learning tools. 

Table: 1 

Participants' perceptions toward the importance levels of technical competences 


Technical Competences 

X 

sd 

To be able to preview the content onstage 

4.21 

1.08 

To be able to upload the content on LMS 

4.20 

1.15 

To have importing and exporting options for different media files 

4.08 

1.05 

To have update options 

4.01 

1.40 

To be able to take screenshot 

3.73 

1.44 

To have Rss and Podcast connections 

3.54 

1.24 

To have mobile application 

3.17 

1.03 

To be able to give time limits, specific to the prepared pages 

3.33 

1.37 

To be able to give links to the content 

3.04 

1.10 

To have mid-term exam preparation options 

3.00 

1.43 

To be able to encode the content 

2.98 

1.34 

Not to have a page-limit for the content 

2.88 

1.08 

To be able to customize screen width when taking screenshot 

2.38 

1.16 

To be able to make changes on the screenshot 

2.17 

1.32 

Total 

3.33 

1.07 


The results shown in the table above indicate that after defining 14 different technical 
competence levels for the e-learning content development tools, participants rated the 
items within the range of moderately important(3) to very important (5). Those which 
rated as very important are 'to be able to preview the content onstage (4.21), to be able 
to upload the content on LMS (4.20), to have importing and exporting options for 
different media files (4.08), and to have update options (4.01). On the other hand, 'to be 
able to take screenshot (3.73), to have Rss and Podcast connections (3.54), to be able to 
give time limits to the prepared pages (3.33), to be able to give links to the content 
(3.04), and to have mid-term exam preparation options' were rated at the level of 
important by the participants. 
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Not perceived as not important though, 'not to have a page-limit for the content (2.88), 
to be able to encode the content (2.98), customize screen width when taking screenshot 
(2.38), and make changes on the screenshot (2.17) were considered as moderately 
important factors of e-learning content development software. The overall importance 
rate of the first group of criteria was measured as 3.33 in the category of important. The 
following table will portray the competences proposed by the participants and their 
ratings of importance for the group of criteria under the heading of 'media competences'. 

Table: 2 

Participants' perceptions toward the importance levels of media competences 


Media Competences 

X 

sd 

To be able to support outputs with various media extensions (swf, jpg, avi, 
mp3, mp4, flv) 

4.33 

1.10 

To be able to modify contrast, brightness, color options of visuals 

4.29 

1.06 

To be able to record audio files 

3.96 

1.10 

To be able to watch videos from different URL addresses 

3.80 

1.16 

To be able to create interactive texts 

3.73 

1.29 

To be able to create animated texts 

3.69 

1.33 

To be able to use zoom area tool 

3.53 

1.24 

To have transition effect features for the prepared pages 

2.76 

1.28 

Total 

3.32 

1.07 


Table: 2 shows the participants' ratings toward the eight different sub-criteria to 
evaluate media competences of e-learning software. The results point out that while two 
of the tools' capabilities related to supporting (4.33) and modifications (4.29) of audio¬ 
visual files were rated as very important, participants declared that to be able to; record 
audio files (3.96), watch videos from different URL addresses (3.80), create interactive 
texts (3.73) and animated texts (3.69), and to be able to use zoom area tool (3.53) are 
also important for evaluating e-learning software. The only item which was regarded as 
moderately important is about whether the tool has transition effect features for the 
prepared pages or not (2.76). The total value dedicated to the media related 
competences of e-learning tools was observed as 3.32 which fall into the 'important' 
category from the perspectives of participants. The following table, including nine 
competences, shows how the participants rated assessment related competences of e- 
learning software. 

Table: 3 

Participants' perceptions toward the importance of assessment competences 


Assessment Competences 

X 

sd 

To be able to give feedbacks to the answers given for the questions 

4.57 

1.14 

To be able to give grades to the answers given in quizzes 

4.30 

1.11 

To be able to write hints about the question to the question area 

4.26 

1.10 

To be able to organize a quiz group 

4.14 

1.06 

To have a question pool option 

3.93 

1.03 

To be able to import questions from exterior question pools 

3.86 

1.07 

To be able to ask questions over pictures 

3.78 

1.17 

To be able to add pictures to the questions 

3.46 

1.01 

To have a voice alert in feedbacks 

2.76 

1.12 

Total 

3.90 

1.11 


According to the results, participants perceived 'to be able to; give feedbacks to the answers given 
for the questions (4.57), give grades to the answers given in quizzes (4.30), write hints about the 
question to the question area (4.26), and to be able organize a quiz group (4.14) as very important 
benchmarks of e-learning software evaluation. 
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The results also indicated that while the only competence rated as moderately important 
is to have a voice alert in feedbacks (2.76), the items related to be able to; import 
questions from exterior question pools (3.86), ask questions over pictures (3.78), add 
pictures to the questions (3.46), and to have a question pool option (3.93) were rated as 
important factors while evaluating e-learning software. The overall rating of assessment 
competences were came out as 3.90 which is the highest competency category 
comparing to the other two groups. The following table is related to participants 
evaluation of specific e-learning tools focused throughout the evaluation phase. The scale 
exploited includes the items of unsatisfactory (1), poor (2), satisfactory (3), good (4), 
and outstanding (5). 

Belve Table: 4 provides results under three sub categories of competencies (technical, 
media, assessment), and total values. In terms of technical competences, all of the 
software were considered as 'good' and 'outstanding' except for 'Ready Go Web (2.49), 
Viewlet Builder (2.76), and Mos Solo (2.34). While Captivate (5.00), Softchalk (5.00), and 
Lectora (4.76) are outstanding group of the list in both media and assessment 
competences, the only tool labelled as 'poor' is Mos Solo (1.79). None of the tools were 
seen as poor or unsatisfactory in terms of assessment competences. The overall values 
point out that Captivate (4.95), Softchalk (4.36), Lectora (4.67) are evaluated as the 
leading competent tools comparing to the rest of the list. Camtasia (3.80), Articulate 
(3.80), and Ignite 4 (3.07) were also labelled as 'good' software. The rest of the tools 
were considered as 'satisfactory' by the participants. In comparison with each other, all 
of the tools have various pros and cons though; none of them was evaluated as 
unsatisfactory. 

Table: 4. 

Participants evaluations of specific e-learning tools in terms of the quality competences 



Technical 

Media 

Assessment 

Total 

X 

SD 

X 

SD 

X 

SD 

X 

SD 

Captivate 

4,86 

1,40 

5,00 

1,87 

5,00 

1,79 

4,95 

1,68 

Softchalk 

3,39 

0,07 

5,00 

1,87 

4,68 

1,47 

4,36 

1,09 

Camtasia Studio 

4,34 

0,88 

3,74 

0,61 

3,34 

0,13 

3,80 

0,53 

Lectora Inspire 

4,23 

0,77 

4,76 

1,63 

5,00 

1,79 

4,67 

1,40 

Articulate 

4,01 

0,55 

3,34 

0,21 

4,03 

0,82 

3,80 

0,53 

Ignite 4 

3,12 

0,34 

3,24 

0,11 

2,87 

0,34 

3,07 

0,20 

Ready Go Web 

2,49 

0,97 

3,05 

0,08 

3,27 

0,06 

2,94 

0,33 

Turbo Demo 

3,07 

0,39 

2,53 

0,60 

3,08 

0,13 

2,90 

0,37 

Viewlet Builder 

2,76 

0,70 

2,37 

0,76 

2,53 

0,68 

2,55 

0,72 

Tool book 

3,67 

0,21 

2,63 

0,50 

2,58 

0,63 

2,96 

0,31 

Tutor Author NG 

3,45 

0,01 

2,48 

0,65 

2,28 

0,93 

2,74 

0,53 

Tranier 

3,58 

0,12 

2,26 

0,87 

2,43 

0,78 

2,75 

0,52 

Content Point 

3,41 

0,05 

2,59 

0,54 

2,67 

0,54 

2,89 

0,38 

Mos Solo 

2,34 

1,12 

1,79 

1,34 

2,12 

1,09 

2,08 

1,19 

Course Lab 

3,28 

0,18 

2,17 

0,96 

2,35 

0,86 

2,60 

0,67 


DISCUSSION AND FINDINGS 

As mentioned in the literature review, among interactive system developers and users 
there is now much agreement that usability is an essential quality of software systems. 
Usability evaluation is a core component of user-centered systems design and an 
essential competency for HCI domain. However, evaluating software for pedagogical 
purposes is not a straightforward job as in evaluating printed materials. The evaluation 
of e-learning applications deserves special attention, and evaluators need effective 
methodologies and appropriate guidelines to perform their task. 
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The present study was designed to determine a usability guideline for e-learning content 
development software. The study is also intended to provide insights to e-learning 
practitioners and distance education stakeholders in terms of how e-learning software 
can empower instructors and learners to develop e-learning specifications to meet 
individual instructional goals. 

The findings from this study make several contributions to the current literature. The 
overall findings of the current study suggested 31 criteria classified under three 
categories as technical (14), media (8), and assessment (9) competence levels for the e- 
learning content development tools. In other words, evaluation patterns that are able to 
lead the educational stakeholders in the analysis of e-learning software are classified 
within technical, media, and assessment related benchmarks. The highly checked options 
for technical competences were related to uploading, previewing, importing, and 
exporting skills of the software. Users also noted that supporting various output file 
types and being able in modifying the visuals are critical media competences. Besides, 
participants underlined that being able to provide an interactive and feedback-rich 
assessment environment is a required feature for effective e-learning instruction through 
the software. Among the groups of benchmarks proposed and rated by the users, 
assessment was considered as the most important one while technical and media 
features were even. 

The results of participants' evaluation of e-learning software revealed that there is a 
threefold discrepancy among the tools evaluated within the sub categories of technical, 
media, and assessment competencies. The first group including Captivate, Softchalk, and 
Lectora were regarded as outstanding tools by the participants. The highest rates were 
observed within the technical competences section. 

The tools which were labeled in second category are Camtasia, Articulate, and Ignite 4. 
There are also tools while rated as satisfactory but not perceived as fully competent. To 
sum up, the e-learning software systems explored have varying functionalities and 
different user interface designs. They might have been built based on the same kind of 
learning processes in mind. Apparently, most tools were observed as having limitations 
in terms of capabilities. Some others have complex functionalities or may become terrific 
for some uses. Furthermore, needs in e-learning environments are likely to change and so 
one tool is unlikely to do everything it is supposed to do. E-learning tools are needed to 
facilitate sophisticated interactivity between the learner and the materials, tutorials, 
assessments, simulations, games, and animated models. 

The usability evaluation process shows that e-learning programmes should be developed 
to be student centered, relevant, motivational, and able to accommodate individual 
student study routines. Besides, they must be well designed web based, educational 
resources. E-learning activities must be prepared to be as flexible as possible. However, 
with a small sample size of tools, caution must be applied, as the findings might not be 
transferable to all e-learning software being currently used in the field. By the time you 
are done compiling a list of e-learning tools, it is likely that there are even more of them 
available then when you started compiling the list. We've mentioned a few of them 
below, but I am sure that my list is less than complete. 

Although graphics tools are commonly used to build buttons, logos, image maps, and 
other such items that are components of online instructional content, they are not 
reviewed here. The current investigation was also limited by the time. More research on 
this topic needs to be undertaken before the association between effective learning and 
the design criteria of software is more clearly understood. 
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